ipayipi

opening the pipeline:
imbibe data

01 February, 2024

Summary

‘ipayipi’ was built for processing time-series data using ‘data.table’ functionality and speed for processing large datasets. This vignette introduces batch processing of time-series data using R ipayipi: covering extraction and standardisation of logger data. Specifically, the five steps outlined in this vignette are:

  1. Load R ipayipi (install from GitHub),
  2. Initiate pipeline,
  3. Imbibe data,
  4. Standardise data, &
  5. Make station files.

An overview of the code to execute these five steps are shown below. This code is all that is required to run each time new data is available. For more details and options for each step, see respective function documentation/help files.

## 1. load pacakge
library(ipayipi)

## 2. initiate pipeline ----
# define our general working directory: wherever data is going to be processed ...
wd <- "ipayipi/data-raw/eg_data/met_eg"

# define directory from which data will be sourced
sd <- "ipayipi/data-raw/eg_data/met_eg/dta_in"

# setup pipeline directory structure
pipe_house <- ipip_init(work_dir = wd, source_dir = sd)

## 3. read & imbibe in data ----
logger_data_import_batch(pipe_house)
imbibe_raw_batch(pipe_house, file_ext_in = ".dat",
  data_setup = ipayipi::cs_toa5)

## 4. standardise data ----
header_sts(pipe_house)
phenomena_sts(pipe_house)

## store standardised data ----
# transfer standardised data to the nomvet_room
transfer_sts_files(pipe_house)

## 5. append standardised data files ----
append_station_batch(pipe_house)

Introduction

Organising time-series data from loggers, such as, weather stations or ground-water sensors, requires no small amount of data structuring. ipayipi helps make this process dynamic and structured, so that, processing, down (and back up) a pipeline, is traceable. Importantly, ipayipi preserves raw-data integrity and initiates archival—so raw data is available in a standardised format—and the processing of this data can be updated.

The package can be found on GitHub and can be installed using the ‘devtools’ package: devtools::install_github("SAEONData/ipayipi").

Initiate pipeline: the ‘pipe_house’

The ‘pipe_house’ is the name of the directory where we will initiate a data-pipeline structure. To keep things simple, we will only use our ‘pipe_house’ for defined data streams. In this vignette the data stream consists of meteorological data gathered from a weather station data logger. ipayipi handles most time-series data formats readable from flat files into R. Data for this vignette can be downloaded from GitHub in the package’s raw-data folder here.

# setting up the 'pipe_house'
## general pipeline working directory
wd <- "ipayipi/data-raw/eg_data/met_eg"

## data souce folder
sd <- "ipayipi/data-raw/eg_data/met_eg/dta_in"

## initiate pipeline
pipe_house <- ipip_init(work_dir = wd, source_dir = sd)
print(pipe_house)
## $wait_room
##                                   wait_room 
## "ipayipi/data-raw/eg_data/met_eg/wait_room" 
## 
## $nomvet_room
##                                   nomvet_room 
## "ipayipi/data-raw/eg_data/met_eg/nomvet_room" 
## 
## $ipip_room
##                                   ipip_room 
## "ipayipi/data-raw/eg_data/met_eg/ipip_room" 
## 
## $raw_room
##                                   raw_room 
## "ipayipi/data-raw/eg_data/met_eg/raw_room" 
## 
## $source_dir
## [1] "ipayipi/data-raw/eg_data/met_eg/dta_in"

What has ipip_init() done? It has created the following directories, if they don’t already exist*:

  1. ‘source_dir’: where new logger data is going to be made available.
  2. ‘wait_room’: waiting room for imbibing data into the pipeline.
  3. ‘nomvet_room’: where standardised/corrected logger files get archived (nomenclature vetted).
  4. ‘ipip_room’: here data get appended into contiguous single station records, and processed.
  5. ‘raw_room’: where ‘unaltered’ raw data gets pushed.

*NB! Running this function will not overwrite existing data.

Imbibe data

In this step, data gets pulled from pipelines data source, that is, the ‘source directory’ (pipe_house$source_dir), into the ‘waiting room’ (pipe_house$wait_room). The example data contains two-years of Cambell Scientific logger text files derived from sensors on a SAEON meteorological station in northern Maputaland, South Africa.

# copy data from source to the wait_room
logger_data_import_batch(pipe_house = pipe_house,
  file_ext = ".dat", # the file extension (with period) of raw data files
  verbose = FALSE, # set to TRUE to report progress in the terminal
  unwanted = "02.2022" # excluding the import of Feb 2022 to make data 'gap'
)

Now that some data is in the ‘wait_room’ directory we can read it into R. Note the pre-set ‘data_setup’ option for Cambell Scientific TOA5 formatted files ipayipi::cs_toa5.

imbibe_raw_batch(pipe_house = pipe_house,
  data_setup = ipayipi::cs_toa5, # standard for reading t0a5 formatted files 
  record_interval_type = "continuous"
)

For more on data input formats, that is, the ‘data_setup’ argument, see the help files of the imbibe_raw_logger_dt() function (i.e., ?imbibe_raw_logger_dt).

Record-interval type is an important parameter. ipayipi handles continuous, event-based (discontinuous), and mixed time-series data types. Record intervals get evaluated using the record_interval_eval() function. Record interval information will be important for further steps, such as, identifying ‘gaps’ or missing data automatically.

Standardise data

Both file-header information, plus other phenomena (variable) metadata, will now be standardised. The spelling/synonyms of file names and associated header metadata have to be scrutinised first. Only after header information gets standardised, can we move on to working on the phenomena. These steps are essential for automating file record appending and downstream data correction/processing (e.g., drift correction).

header_sts(pipe_house)

If it is the first-time running header_sts(), or new synonyms get introduced into pipe-house directory, header_sts() will produce a warning. This is because the user needs to define new nomenclature standards. Unstandardised names (or columns) have the preffix ‘uz. These standards get stored in a file called ‘nomtab.rns’ in the ‘waiting room’. If this file is deleted—a new one will be generated—but the user will have to populate the tables with synonym vocab.

The nomenclature table in the ‘waiting room’ can be updated from ‘csv’ format (or directly in R). If a new synonym gets introduced—the file containing new nomenclature will be skipped in further processing—a ‘csv’ version of the ‘nomtab.rns’ will be copied to the ‘waiting room’ for editing.

Only the following fields — with NAs — require editing in the ‘nomtab’ ‘csv’:

pt <- read.csv("ipayipi/data-raw/eg_data/met_eg/nomtab_display.csv")
kbl(pt) |>
  kable_paper("hover") |>
  kable_styling(font_size = 11) |>
  column_spec(c(1, 7:8, 11), background = "#df9a86") |>
  column_spec(c(2:6, 9:10, 12), background = "#aed8f0") |>
  scroll_box(width = "100%", height = "400px")
uz_station location station stnd_title logger_type logger_title uz_record_interval_type uz_record_interval record_interval_type record_interval uz_table_name table_name
Mabasa AWS mcp mabasa_aws mcp_mabasa_aws CR1000X CR1000 mixed 1_days mixed 1_days TableDay raw_1_days
Mabasa AWS mcp mabasa_aws mcp_mabasa_aws CR1000X CR1000 mixed 1_hours mixed 1_hours TableHour raw_1_hours
Mabasa AWS mcp mabasa_aws mcp_mabasa_aws CR1000X CR1000X continuous 1_days continuous 1_days TableDay raw_1_days
Mabasa AWS mcp mabasa_aws mcp_mabasa_aws CR1000X CR1000X continuous 1_hours continuous 1_hours TableHour raw_1_hours
Station2 mcp mabasa_aws mcp_mabasa_aws CR1000X CR1000X continuous 1_days continuous 1_days TableDay raw_1_days
Station2 mcp mabasa_aws mcp_mabasa_aws CR1000X CR1000X continuous 1_hours continuous 1_hours TableHour raw_1_hours
Sibayi AWS mcp sibayi_aws mcp_sibayi_aws CR200X CR200X continuous 1_days continuous 1_days Daily raw_1_days
Sibayi AWS mcp sibayi_aws mcp_sibayi_aws CR200X CR200X continuous 5_mins continuous 5_mins Five_Minutes raw_5_mins
Sibayi AWS mcp sibayi_aws mcp_sibayi_aws CR200X CR200X continuous 1_hours continuous 1_hours Hourly_ raw_1_hours
84923 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 1_days continuous 1_days Daily raw_1_days
84923 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 5_mins continuous 5_mins Five_mins raw_5_mins
84923 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 1_hours continuous 1_hours Hourly raw_1_hours
88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 1_days continuous 1_days Daily raw_1_days
88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 5_mins continuous 5_mins Five_mins raw_5_mins
88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 1_hours continuous 1_hours Hourly raw_1_hours
CR1000 - Vasi_Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 1_days continuous 1_days Daily raw_1_days
CR1000 - Vasi_Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 5_mins continuous 5_mins Five_mins raw_5_mins
CR1000 - Vasi_Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 1_hours continuous 1_hours Hourly raw_1_hours
CR1000_Vasi Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 1_days continuous 1_days Daily raw_1_days
CR1000_Vasi Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous discnt continuous discnt Daily raw_1_days
CR1000_Vasi Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 5_mins continuous 5_mins Five_mins raw_5_mins
CR1000_Vasi Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 1_hours continuous 1_hours Hourly raw_1_hours
Science Center AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 1_days continuous 1_days Daily raw_1_days
Science Center AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 5_mins continuous 5_mins Five_mins raw_5_mins
Science Center AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 1_hours continuous 1_hours Hourly raw_1_hours
Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 1_days continuous 1_days Daily raw_1_days
Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 5_mins continuous 5_mins Five_mins raw_5_mins
Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 mixed 5_mins mixed 5_mins Five_mins raw_5_mins
Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws CR1000 CR1000 continuous 1_hours continuous 1_hours Hourly raw_1_hours
CR200Series mcp sibayi_aws mcp_sibayi_aws CR200X NA continuous 1_days continuous 1_days Daily raw_1_days
CR200Series mcp sibayi_aws mcp_sibayi_aws CR200X NA continuous 5_mins continuous 5_mins Five_Minutes raw_5_mins
CR200Series mcp sibayi_aws mcp_sibayi_aws CR200X NA continuous 1_hours continuous 1_hours Hourly_ raw_1_hours
Sibayi mcp sibayi_aws mcp_sibayi_aws CR300 NA continuous 1_hours continuous 1_hours Table1 raw_1_hours
Sibayi mcp sibayi_aws mcp_sibayi_aws CR300 NA continuous 1_days continuous 1_days Table2 raw_1_days
Sibayi mcp sibayi_aws mcp_sibayi_aws CR300 NA continuous 5_mins continuous 5_mins Table3 raw_5_mins
Sibayi AWS mcp sibayi_aws mcp_sibayi_aws CR200X NA continuous discnt continuous discnt Daily raw_1_days
Sibayi Camp Mini AWS mcp sibayi_aws mcp_sibayi_aws CR200X NA mixed 1_days mixed 1_days Daily raw_1_days
Sibayi Camp Mini AWS mcp sibayi_aws mcp_sibayi_aws CR200X NA continuous 1_days continuous 1_days Daily raw_1_days
Sibayi Camp Mini AWS mcp sibayi_aws mcp_sibayi_aws CR200X NA continuous 5_mins continuous 5_mins Five_Minutes raw_5_mins
Sibayi Camp Mini AWS mcp sibayi_aws mcp_sibayi_aws CR200X NA continuous 1_hours continuous 1_hours Hourly_ raw_1_hours
Sibayi Mini AWS mcp sibayi_aws mcp_sibayi_aws CR200X NA continuous 1_days continuous 1_days Daily raw_1_days
Sibayi Mini AWS mcp sibayi_aws mcp_sibayi_aws CR200X NA continuous 5_mins continuous 5_mins Five_Minutes raw_5_mins
Sibayi Mini AWS mcp sibayi_aws mcp_sibayi_aws CR200X NA continuous 1_hours continuous 1_hours Hourly_ raw_1_hours
Sibayi Mini Station mcp sibayi_aws mcp_sibayi_aws CR200X NA continuous 1_days continuous 1_days Daily raw_1_days
Sibayi Mini Station mcp sibayi_aws mcp_sibayi_aws CR200X NA continuous 5_mins continuous 5_mins Five_Minutes raw_5_mins
Sibayi Mini Station mcp sibayi_aws mcp_sibayi_aws CR200X NA continuous 1_hours continuous 1_hours Hourly_ raw_1_hours

Once NA values of the above fields have been populated the edited ‘csv’ will be imbibed into the pipeline structure when rerunning header_sts(pipe_house)—this function will imbibe the most recently updated ‘csv’ nomenclature table from the ‘wait_room’ into the pipeline, and standardised header nomenclature.

In step with good tidy data standards, keep nomenclature to ‘snake case’ with no special characters (bar the useful underscore’). ’ Standardising phenomena metadata follows a similar process as for header-data standardisation. If the phenomena standards have been described and there is a ‘phentab.rps’ in the ‘waiting room’, running the below code updates all files phenomena details.

phenomena_sts(pipe_house = pipe_house, unwanted = "display")

If there is no ‘phenomena table’ (‘phentab.rps’), one NA values in the ‘csv’ copy need to be described. The following fields in the ‘csv’ phentab must be populated:

Additional fields that are not mandatory include:

If an ‘f_convert’ factor (scroll right on the table below) is applied to phenomena, the standardised units must be different from the unstandardised units (uz_units) in the phenomena table. This ensures that phenomena that are appended have similar units.

After filling in details, to replace NA values, rerun phenomena_sts(pipe_house), to imbibe the updated phenomena descriptions, and update the logger data being standardised.

Standardised data files get transferred to the ‘nomenclature vetted’ directory (‘nomtab room’) using the function below. After being transferred, files in the waiting room (except the nomtab and phentab standards) are automatically removed.

# move standardised files to a storage directory
transfer_sts_files(pipe_house)

Archiving raw data files: Before removing raw unstandardised files—if there is a ‘raw_room’ directory in the pipeline working directory—raw input data files will be copied to this directory and filed in folders by year and month of the lasted date of recording. This is done by the imbibe_raw_batch() function.

Make station files

The append_station_batch() function updates station files in the ‘ipip_room’ with files from the ‘nomvet_room’.

# append station files + metadata records
# note the 'cores' argument --- parallel processing supported on Linux systems
append_station_batch(pipe_house, cores = 2)

Now that a station file has been generated for the Vasi Science Centre weather station we can check what tables have been created/appended. Station files are maintained in the ‘ipip_room’ of the pipeline’s folder structure.

# list station files in the ipip directory
sf <- dta_list(
  input_dir = pipe_house$ipip_room, # search directory
  file_ext = ".ipip", # note the station's default file extension
)

# check what stations are in the ipip room
print(sf)
## [1] "mcp_vasi_science_centre_aws.ipip"
# read in the station file
sf <- readRDS(file.path(pipe_house$ipip_room, sf[1]))

# names of the tables stored in the station file
names(sf)
## [1] "data_summary"      "gaps"              "logg_interfere"    "phen_data_summary" "phens"             "raw_1_days"        "raw_1_hours"       "raw_5_mins"

‘Raw’ data

Our station file has three ‘raw’ data tables, with 5 minute, daily, and monthly data.

# a look 1st data row of 5 minute data
print(sf$raw_5_min[1, ])
##    id           date_time wind_speed wind_dir wind_dir_sd temp_air_avg humid_rel_pcnt solar_rad_avg uv_rad_avg rain_tot temp_ground_avg
## 1:  0 2020-06-11 12:35:00       3.29    309.2       42.72        31.27          38.04         584.1      24.86        0           32.92
# use kableExtra to view first 20 data rows --- default printing of tables doesn't
#  looks good in html
kbl(sf$raw_5_min[1:20, ]) |>
  kable_paper("hover") |>
  kable_styling(font_size = 11) |>
  scroll_box(width = "100%", height = "400px")
id date_time wind_speed wind_dir wind_dir_sd temp_air_avg humid_rel_pcnt solar_rad_avg uv_rad_avg rain_tot temp_ground_avg
0 2020-06-11 12:35:00 3.290 309.2 42.72 31.27 38.04 584.1 24.86 0 32.92
1 2020-06-11 12:40:00 3.722 338.0 27.10 30.89 38.86 582.1 24.67 0 32.69
2 2020-06-11 12:45:00 3.897 322.8 19.89 30.95 39.30 578.1 24.41 0 32.58
3 2020-06-11 12:50:00 3.545 340.0 28.47 30.91 38.91 572.1 24.00 0 32.45
4 2020-06-11 12:55:00 4.050 333.9 25.85 30.92 39.44 566.3 23.60 0 32.47
5 2020-06-11 13:00:00 3.847 344.2 31.73 30.89 38.24 556.8 22.95 0 32.55
6 2020-06-11 13:05:00 3.442 334.3 32.61 30.96 37.96 553.2 22.87 0 32.63
7 2020-06-11 13:10:00 3.085 332.0 28.55 31.17 37.76 545.6 22.51 0 32.63
8 2020-06-11 13:15:00 3.772 326.0 37.08 31.31 37.47 539.7 22.05 0 32.83
9 2020-06-11 13:20:00 3.795 343.7 32.37 31.07 37.19 528.5 21.49 0 32.74
10 2020-06-11 13:25:00 3.403 337.0 28.52 31.22 36.62 521.3 21.38 0 32.70
11 2020-06-11 13:30:00 3.165 337.8 38.35 31.41 35.68 516.0 21.28 0 32.81
12 2020-06-11 13:35:00 3.307 334.6 32.02 31.51 36.00 507.3 20.91 0 32.97
13 2020-06-11 13:45:00 3.006 348.9 21.52 31.45 35.30 491.1 20.20 0 32.98
14 2020-06-11 13:55:00 2.538 326.6 0.00 31.43 34.45 473.1 19.42 0 32.74
15 2020-06-11 14:00:00 3.470 331.2 27.09 31.58 34.13 467.7 19.18 0 32.77
16 2020-06-11 14:05:00 3.690 326.4 36.23 31.61 34.41 454.5 18.53 0 32.76
17 2020-06-11 14:10:00 2.800 339.9 24.61 32.08 33.83 429.4 17.54 0 33.21
18 2020-06-11 14:15:00 3.100 341.2 40.78 31.85 34.96 422.1 17.27 0 33.06
19 2020-06-11 14:20:00 3.613 332.9 32.83 31.67 34.00 411.2 16.69 0 32.80

The ‘raw’ data-header summary

This table contains summary information on the origin of each data file used to make up the station file.

# using kableExtra
kbl(sf$data_summary) |> kable_paper("hover") |> kable_styling(font_size = 11) |>
  scroll_box(width = "100%", height = "400px")
dsid file_format uz_station location station stnd_title start_dttm end_dttm logger_type logger_title logger_sn logger_os logger_program_name logger_program_sig uz_record_interval_type uz_record_interval record_interval_type record_interval uz_table_name table_name nomvet_name file_origin
1 TOA5 88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2020-06-12 00:00:00 2021-02-10 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20200612-20210210__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20210210__1.dat
2 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2020-12-09 00:00:00 2021-01-12 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20201209-20210112__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily12.01.2021__1.dat
3 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-01-13 00:00:00 2021-04-14 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20210113-20210414__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily14.04.2021__1.dat
4 TOA5 88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-02-11 00:00:00 2021-03-09 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20210211-20210309__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20210309__1.dat
5 TOA5 88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-03-10 00:00:00 2021-05-12 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20210310-20210512__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20210512__1.dat
6 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-05-13 00:00:00 2021-06-16 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20210513-20210616__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily16.06.2021__1.dat
7 TOA5 88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-05-13 00:00:00 2021-07-21 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20210513-20210721__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20210721__1.dat
8 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-06-17 00:00:00 2021-08-11 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20210617-20210811__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily11.08.2021__1.dat
9 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-08-12 00:00:00 2021-09-13 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20210812-20210913__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily13.09.2021__1.dat
10 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-09-14 00:00:00 2021-10-13 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20210914-20211013__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily13.10.2021__1.dat
11 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-10-14 00:00:00 2021-11-08 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20211014-20211108__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily16.06.2021__2.dat
12 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-11-09 00:00:00 2021-12-07 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20211109-20211207__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily07.12.2021__1.dat
13 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-12-08 00:00:00 2022-01-12 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20211208-20220112__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily07.12.2021__2.dat
14 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-02-10 00:00:00 2022-03-07 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20220210-20220307__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily07.03.2022__1.dat
15 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-03-08 00:00:00 2022-04-12 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20220308-20220412__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20220412__1.dat
16 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-04-13 00:00:00 2022-05-11 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20220413-20220511__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20220511__1.dat
17 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-05-12 00:00:00 2022-06-13 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20220512-20220613__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Daily_20220613__1.dat
18 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-06-14 00:00:00 2022-07-12 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20220614-20220712__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily07.12.2021__3.dat
19 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-07-13 00:00:00 2022-08-10 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20220713-20220810__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily07.12.2021__4.dat
20 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-08-11 00:00:00 2022-09-15 00:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_days continuous 1_days Daily raw_1_days mcp_vasi_science_centre_aws_1_days_20220811-20220915__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Daily07.12.2021__5.dat
21 TOA5 88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2020-06-11 13:00:00 2021-02-10 14:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20200611-20210210__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20210210__1.dat
22 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2020-12-08 15:00:00 2021-01-12 11:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20201208-20210112__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly12.01.2021__1.dat
23 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-01-12 12:00:00 2021-04-14 13:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20210112-20210414__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__1.dat
24 TOA5 88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-02-10 15:00:00 2021-03-09 14:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20210210-20210309__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20210309__1.dat
25 TOA5 88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-03-09 15:00:00 2021-05-12 14:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20210309-20210512__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20210512__1.dat
27 TOA5 88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-05-12 15:00:00 2021-07-21 13:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20210512-20210721__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20210721__1.dat
26 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-05-12 16:00:00 2021-06-16 11:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20210512-20210616__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly16.06.2021__1.dat
28 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-06-16 12:00:00 2021-08-11 15:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20210616-20210811__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly11.08.2021__1.dat
29 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-08-11 16:00:00 2021-09-13 15:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20210811-20210913__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly13.09.2021__1.dat
30 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-09-13 16:00:00 2021-10-13 15:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20210913-20211013__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly13.10.2021__1.dat
31 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-10-13 16:00:00 2021-11-08 12:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20211013-20211108__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__2.dat
32 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-11-08 13:00:00 2021-12-07 14:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20211108-20211207__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__3.dat
33 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-12-07 15:00:00 2022-01-12 11:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20211207-20220112__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__4.dat
34 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-02-09 17:00:00 2022-03-07 16:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20220209-20220307__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly07.03.2022__1.dat
35 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-03-07 17:00:00 2022-04-12 14:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20220307-20220412__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20220412__1.dat
36 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-04-12 15:00:00 2022-05-11 14:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20220412-20220511__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20220511__1.dat
37 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-05-11 15:00:00 2022-06-13 13:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20220511-20220613__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Hourly_20220613__1.dat
38 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-06-13 14:00:00 2022-07-12 15:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20220613-20220712__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__5.dat
39 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-07-12 16:00:00 2022-08-10 15:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20220712-20220810__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__6.dat
40 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-08-10 16:00:00 2022-09-15 11:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 1_hours continuous 1_hours Hourly raw_1_hours mcp_vasi_science_centre_aws_1_hours_20220810-20220915__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Hourly14.04.2021__7.dat
41 TOA5 88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2020-06-11 12:35:00 2021-02-10 14:45:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20200611-20210210__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20210210__1.dat
42 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2020-12-08 14:55:00 2021-01-12 11:35:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20201208-20210112__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins12.01.2021__1.dat
43 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-01-12 11:40:00 2021-04-14 13:40:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20210112-20210414__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__1.dat
44 TOA5 88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-02-10 14:50:00 2021-03-09 14:30:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20210210-20210309__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20210309__1.dat
45 TOA5 88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-03-09 14:35:00 2021-05-12 14:55:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20210309-20210512__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20210512__1.dat
47 TOA5 88296 mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-05-12 15:00:00 2021-07-21 13:45:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20210512-20210721__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20210721__1.dat
46 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-05-12 15:20:00 2021-06-16 11:50:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20210512-20210616__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins16.06.2021__1.dat
48 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-06-16 11:55:00 2021-08-11 15:20:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20210616-20210811__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins11.08.2021__1.dat
49 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-08-11 15:25:00 2021-09-13 15:45:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20210811-20210913__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins13.09.2021__1.dat
50 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-09-13 15:50:00 2021-10-13 15:40:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20210913-20211013__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins13.10.2021__1.dat
51 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-10-13 15:45:00 2021-11-08 12:55:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20211013-20211108__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__2.dat
52 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-11-08 13:00:00 2021-12-07 14:40:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20211108-20211207__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__3.dat
53 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2021-12-07 14:45:00 2022-01-12 11:55:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20211207-20220112__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__4.dat
54 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-02-09 16:10:00 2022-03-07 16:10:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20220209-20220307__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins07.03.2022__1.dat
55 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-03-07 16:15:00 2022-04-12 14:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20220307-20220412__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20220412__1.dat
56 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-04-12 14:05:00 2022-05-11 14:35:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20220412-20220511__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20220511__1.dat
57 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-05-11 14:40:00 2022-06-13 13:35:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20220511-20220613__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/ScienceCentreAWS_Five_mins_20220613__1.dat
58 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-06-13 13:40:00 2022-07-12 15:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20220613-20220712__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__5.dat
59 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-07-12 15:05:00 2022-08-10 15:00:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20220712-20220810__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__6.dat
60 TOA5 Science Centre AWS mcp vasi_science_centre_aws mcp_vasi_science_centre_aws 2022-08-10 15:05:00 2022-09-15 11:45:00 CR1000 NA 88296 CR1000.Std.31.08 CPU:Vasi_Science Centre AWS.CR1 64103 continuous 5_mins continuous 5_mins Five_mins raw_5_mins mcp_vasi_science_centre_aws_5_mins_20220810-20220915__1.ipi ipayipi/data-raw/eg_data/met_eg/dta_in/Science Centre AWS_Five_mins14.04.2021__7.dat

The phenomena table: ‘phens’

A station file version of phenomena standards. Note each phenomena variation/synonym has a unique identifier (‘phid’) within the scope of this station.

# using kableExtra
kbl(sf$phens) |> kable_paper("hover") |> kable_styling(font_size = 11) |>
  scroll_box(width = "100%", height = "400px")
phid phen_name_full phen_type phen_name units measure offset var_type uz_phen_name uz_units uz_measure f_convert sensor_id notes table_name origin
1 Humidity, relative: maximum Humidity, relative humid_rel_max_pcnt pcnt max 0 num RH_Max % Max NA NA NA raw_1_days s_f
2 Humidity, relative: minimum Humidity, relative humid_rel_min pcnt min 0 num RH_Min % Min NA NA NA raw_1_days s_f
3 Rainfall: total Rainfall rain_tot mm tot 0 num Rain_mm_Tot mm Tot NA NA NA raw_1_days s_f
4 Solar radiation: maximum Solar radiation solar_rad_max w_per_m2 max 0 num SlrW_Max W/m^2 Max NA NA NA raw_1_days s_f
5 Solar radiation: standard deviation Solar radiation solar_rad_sd w_per_m2 sd 0 num SlrW_Std W/m^2 Std NA NA NA raw_1_days s_f
6 Temperature, air: maximum Temperature, air temp_air_max deg_c max 0 num AirTC_Max Deg C Max NA NA NA raw_1_days s_f
7 Temperature, air: minimum Temperature, air temp_air_min deg_c min 0 num AirTC_Min Deg C Min NA NA NA raw_1_days s_f
8 Temperature, ground level: average Temperature, ground level temp_ground_avg deg_c avg 0 num T107_C_Avg Deg C Avg NA NA NA raw_1_days s_f
9 Temperature, ground level: minimum Temperature, ground level temp_ground_min deg_c min 0 num T107_C_Min Deg C Min NA NA NA raw_1_days s_f
10 UV radiation: maximum UV radiation uv_rad_max w_per_m2 max 0 num CUV5_W_Max W/m^2 Max NA NA NA raw_1_days s_f
11 UV radiation: standard deviation UV radiation uv_rad_sd w_per_m2 sd 0 num CUV5_W_Std W/m^2 Std NA NA NA raw_1_days s_f
12 UV radiation: total UV radiation uv_rad_tot mj_per_m2 tot 0 num CUV5_MJ_Tot MJ/m^2 Tot NA NA NA raw_1_days s_f
13 Wind direction: sample Wind direction wind_dir deg smp 0 num WindDir_D1_WVT Deg WVc NA NA NA raw_1_days s_f
14 Wind direction: standard deviation Wind direction wind_dir_sd deg sd 0 num WindDir_SD1_WVT Deg WVc NA NA NA raw_1_days s_f
15 Wind speed: average Wind speed wind_speed_avg m_per_sec avg 0 num VW_Avg Avg NA NA NA raw_1_days s_f
16 Wind speed: maximum Wind speed wind_speed_max m_per_sec max 0 num WS_ms_Max meters/second Max NA NA NA raw_1_days s_f
17 Wind speed: sample Wind speed wind_speed m_per_sec smp 0 num WS_ms_S_WVT meters/second WVc NA NA NA raw_1_days s_f
18 Battery level: minimum voltage Battery level batt_min_v v min 0 num BattV_Min Volts Min NA NA NA raw_1_hours s_f
19 Humidity, relative: percent Humidity, relative humid_rel_pcnt pcnt smp 0 num RH % Smp NA NA NA raw_1_hours s_f
20 Pressure, atmosphere: sample Pressure, atmosphere pressure_atm kpa smp 0 num BP_kPa kPa Smp NA NA NA raw_1_hours s_f
3 Rainfall: total Rainfall rain_tot mm tot 0 num Rain_mm_Tot mm Tot NA NA NA raw_1_hours s_f
22 Solar radiation: average Solar radiation solar_rad_avg w_per_m2 avg 0 num SlrW_Avg W/m^2 Avg NA NA NA raw_1_hours s_f
23 Temperature, air: average Temperature, air temp_air_avg deg_c avg 0 num AirTC_Avg Deg C Avg NA NA NA raw_1_hours s_f
8 Temperature, ground level: average Temperature, ground level temp_ground_avg deg_c avg 0 num T107_C_Avg Deg C Avg NA NA NA raw_1_hours s_f
9 Temperature, ground level: minimum Temperature, ground level temp_ground_min deg_c min 0 num T107_C_Min Deg C Min NA NA NA raw_1_hours s_f
26 UV radiation: average UV radiation uv_rad_avg w_per_m2 avg 0 num CUV5_W_Avg W/m^2 Avg NA NA NA raw_1_hours s_f
13 Wind direction: sample Wind direction wind_dir deg smp 0 num WindDir_D1_WVT Deg WVc NA NA NA raw_1_hours s_f
14 Wind direction: standard deviation Wind direction wind_dir_sd deg sd 0 num WindDir_SD1_WVT Deg WVc NA NA NA raw_1_hours s_f
15 Wind speed: average Wind speed wind_speed_avg m_per_sec avg 0 num VW_Avg Avg NA NA NA raw_1_hours s_f
17 Wind speed: sample Wind speed wind_speed m_per_sec smp 0 num WS_ms_S_WVT meters/second WVc NA NA NA raw_1_hours s_f
19 Humidity, relative: percent Humidity, relative humid_rel_pcnt pcnt smp 0 num RH % Smp NA NA NA raw_5_mins NA
3 Rainfall: total Rainfall rain_tot mm tot 0 num Rain_mm_Tot mm Tot NA NA NA raw_5_mins NA
22 Solar radiation: average Solar radiation solar_rad_avg w_per_m2 avg 0 num SlrW_Avg W/m^2 Avg NA NA NA raw_5_mins NA
23 Temperature, air: average Temperature, air temp_air_avg deg_c avg 0 num AirTC_Avg Deg C Avg NA NA NA raw_5_mins NA
8 Temperature, ground level: average Temperature, ground level temp_ground_avg deg_c avg 0 num T107_C_Avg Deg C Avg NA NA NA raw_5_mins NA
26 UV radiation: average UV radiation uv_rad_avg w_per_m2 avg 0 num CUV5_W_Avg W/m^2 Avg NA NA NA raw_5_mins NA
13 Wind direction: sample Wind direction wind_dir deg smp 0 num WindDir_D1_WVT Deg WVc NA NA NA raw_5_mins NA
14 Wind direction: standard deviation Wind direction wind_dir_sd deg sd 0 num WindDir_SD1_WVT Deg WVc NA NA NA raw_5_mins NA
17 Wind speed: sample Wind speed wind_speed m_per_sec smp 0 num WS_ms_S_WVT meters/second WVc NA NA NA raw_5_mins NA

Note the ‘phid’ link in the temporal phenomena summary below.

# using kableExtra
kbl(sf$phen_data_summary) |> kable_paper("hover") |> kable_styling(font_size = 11) |>
  scroll_box(width = "100%", height = "400px")
phid phen_name start_dttm end_dttm table_name
1 humid_rel_max_pcnt 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
2 humid_rel_min 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
3 rain_tot 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
4 solar_rad_max 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
5 solar_rad_sd 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
6 temp_air_max 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
7 temp_air_min 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
8 temp_ground_avg 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
9 temp_ground_min 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
10 uv_rad_max 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
11 uv_rad_sd 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
12 uv_rad_tot 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
13 wind_dir 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
14 wind_dir_sd 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
17 wind_speed 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
15 wind_speed_avg 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
16 wind_speed_max 2020-06-12 00:00:00 2022-09-15 00:00:00 raw_1_days
18 batt_min_v 2020-06-11 13:00:00 2022-09-15 11:00:00 raw_1_hours
19 humid_rel_pcnt 2020-06-11 13:00:00 2022-09-15 11:00:00 raw_1_hours
20 pressure_atm 2020-06-11 13:00:00 2022-09-15 11:00:00 raw_1_hours
3 rain_tot 2020-06-11 13:00:00 2022-09-15 11:00:00 raw_1_hours
22 solar_rad_avg 2020-06-11 13:00:00 2022-09-15 11:00:00 raw_1_hours
23 temp_air_avg 2020-06-11 13:00:00 2022-09-15 11:00:00 raw_1_hours
8 temp_ground_avg 2020-06-11 13:00:00 2022-09-15 11:00:00 raw_1_hours
9 temp_ground_min 2020-06-11 13:00:00 2022-09-15 11:00:00 raw_1_hours
26 uv_rad_avg 2020-06-11 13:00:00 2022-09-15 11:00:00 raw_1_hours
13 wind_dir 2020-06-11 13:00:00 2022-09-15 11:00:00 raw_1_hours
14 wind_dir_sd 2020-06-11 13:00:00 2022-09-15 11:00:00 raw_1_hours
17 wind_speed 2020-06-11 13:00:00 2022-09-15 11:00:00 raw_1_hours
15 wind_speed_avg 2020-06-11 13:00:00 2022-09-15 11:00:00 raw_1_hours
19 humid_rel_pcnt 2020-06-11 12:35:00 2022-09-15 11:45:00 raw_5_mins
3 rain_tot 2020-06-11 12:35:00 2022-09-15 11:45:00 raw_5_mins
22 solar_rad_avg 2020-06-11 12:35:00 2022-09-15 11:45:00 raw_5_mins
23 temp_air_avg 2020-06-11 12:35:00 2022-09-15 11:45:00 raw_5_mins
8 temp_ground_avg 2020-06-11 12:35:00 2022-09-15 11:45:00 raw_5_mins
26 uv_rad_avg 2020-06-11 12:35:00 2022-09-15 11:45:00 raw_5_mins
13 wind_dir 2020-06-11 12:35:00 2022-09-15 11:45:00 raw_5_mins
14 wind_dir_sd 2020-06-11 12:35:00 2022-09-15 11:45:00 raw_5_mins
17 wind_speed 2020-06-11 12:35:00 2022-09-15 11:45:00 raw_5_mins

Phenomena append notes: When appending phenomena tables, if there is overlapping data, ‘ipayipi’ will examine each overlapping phenomena series in turn and overwrite either new data (e.g., additions to a station file) or the station file data. Missing data (NAs) will however not be overwritten. The phenomena data summary keeps a temporal record of how phenomena have been appended. Maintaining these records helps data processing down the pipeline.

Missing data—gaps

Checking for data ‘gaps’ in continuous data streams can be fairly straight forward—just highlight the missing/NA values. But with discontinuous or event based data things are more nuanced. gap_eval_batch() identifies gap periods, specifically where a logger was not recording, in continuous and discontinuous time-series data. Here the data is continuous so identifying gaps is simple.

gap_eval_batch(pipe_house, cores = 3)
# read in the station file
sf <- readRDS(file.path("ipayipi/data-raw/eg_data/met_eg",
  "ipip_room/mcp_vasi_science_centre_aws.ipip"))
sf$gaps
##    gid euid gap_type   phen  table_name           gap_start             gap_end    dt_diff_s gap_problem_thresh_s problem_gap notes
## 1:  13   NA     auto logger  raw_1_days 2022-01-12 00:00:00 2022-02-10 00:00:00 2505600 secs                86400        TRUE  <NA>
## 2:  13   NA     auto logger raw_1_hours 2022-01-12 11:00:00 2022-02-09 17:00:00 2440800 secs                 3600        TRUE  <NA>
## 3:  13   NA     auto logger  raw_5_mins 2022-01-12 11:55:00 2022-02-09 16:10:00 2434500 secs                  300        TRUE  <NA>

Note the table and graph correctly show that February 2022 is missing in all raw data tables—we omitted importing this data using the ‘unwanted’ parameter in the import data stage above. Hover over the graph—interact.

p <- plot_raw_availability(pipe_house)
p <- p$plot_availability + scale_colour_sunset(discrete = TRUE) +
      labs(color = "Station") +
      theme(legend.position = "none")
plotly::ggplotly(p)